Experiment Result: Two Phase Flow

Updates

1. Debugged inference method
2. With the updated inference method, did hyperparameter search, especially with learning rate, \(\lambda\)
3. We were wondering why it seemed like there was not so much improvement.
4. We retrained the model with updated eigenvector -> refer to Updated Experiment Setting
5. Loss decreases faster, posterior

Previous Experiment Setting

1. Dataset
   • \(2000\) pairs of \(\{K, S^t(K)\}_{t=1}^8\).
   • Train Test split: [1800, 200]
2. FIM
   • Number of observation = 10
   • Number of eigenvector = 1
   • For a single pair of datapoint, we obtain 1 FIM.
     • Likelihood is difference between perturbed time series of \(\{S^t(K)\}_{t=1}^8\) with true time series of \(\{S^t(K)\}_{t=1}^8\).
3. Hyperparameter
   • Batchsize = 100

Updated Experiment Setting

1. Dataset
   • \(1000\) pairs of \(\{K, S^t(K)\}_{t=1}^8\).
   • Train Test split: [800, 200]
2. FIM
   • Number of observation = 2
   • Number of eigenvector = 1
   • For a single pair of datapoint, we obtain 8 FIM as there are 8 different time steps.
     • Likelihood is difference between perturbed single Saturation \(S^t(K)\) with true singe time step Saturation.
3. Hyperparameter
   • Batchsize = 100

Pipeline

• FNO-NF.jl: create two-phase flow dataset, eigenvector of FIM, and vJp
• Diff_MultiPhysics: train (written in pytorch) and posterior estimation

Training Result

To evaluate training result, we go over three things:

• Loss Behavior
• Forward Simulation
• Posterior Estimation

Loss/Learning Behavior

All loss

Only GM Term

Figure 1: Example Loss plots (GM model:3rd row of the loss table)

	Epochs	\(\lambda\)	Train Loss	Test Loss
			MSE/GM	MSE
FNO-MSE	1000	N.A.	\(3.3622 \times 10^{-8}\)	\(8.4016 \times 10^{-8}\)
FNO-PBI	2000	150.0	\(1.0436 \times 10^{-7}\)	\(1.044 \times 10^{-7}\)
FNO-PBI	2000	1.0	\(3.0028 \times 10^{-8}\)	\(8.0099 \times 10^{-8}\)
FNO-PBI	1000	150.0	\(2.6428 \times 10^{-7}\)	\(1.5976 \times 10^{-7}\)
FNO-PBI	1000	20.0	\(6.2106 \times 10^{-8}\)	\(9.2973 \times 10^{-8}\)
FNO-PBI	1000	5.0	\(6.3265 \times 10^{-8}\)	\(9.7524 \times 10^{-8}\)
FNO-PBI	1000	1.0	\(4.1154 \times 10^{-8}\)	\(8.6791 \times 10^{-8}\)
FNO-PBI	1000	0.7	\(4.2235 \times 10^{-8}\)	\(9.1102 \times 10^{-8}\)

Updated loss table

	Epochs	\(\lambda\)	Train Loss	Test Loss
			MSE/GM	MSE
FNO-MSE	1000	1.0	\(6.5207 \times 10^{-8}\)	\(1.3088 \times 10^{-7}\)
FNO-PBI	1000	1.0	\(8.3925 \times 10^{-8}\)	\(1.3030\times 10^{-7}\)

We now evaluate surrogate models in two different criteria, forward simulation and inverse problem.

Evaluation: Forward Simulation

Forward Simulation on test sample

True Saturation

Predicted Saturation: MSE

Predicted Saturation: PBI

Figure 2: Example of Forward Prediction

Test Sample 1

Test Sample 2

Test Sample 3

Figure 3: Absolute Difference plot of test samples

Learned and True vjp (sanity check)

We observe that for MSE,

1. Scale in the color bar does not match.
2. The learned vjp looks noisy as there are some colors showing in the part where it should be just white.

But in PBI, we verify that the learned vjp and true vjp matches well.

1. The scale of color bar matches correctly.
2. The plot does not look noisy.

True vjp

Learned vjp: MSE

Learned vjp: PBI

Absolute Difference: MSE

Absolute Difference: PBI

What other things can be evaluated in terms of forward simulation?

1. Stability: predict longer saturation evolution 9th to 16th.
2. Generalization: test with out of distribution test samples.

Toy example to test generalizability of MSE and PBI

Also, generated out of distribution samples:

out of distribution: K

(a) In distribution K

Figure 4: In distribution K and out of distribution K

OOD: S

OOD: MSE Forward Prediction

OOD: PBI Forward Prediction

OOD: diff btw learned posterior and K

Things to change in experiment for better forward prediction of PBI

But before conducting all those experiments, we might want to change the eigenvector. That is,

• From single eigenvector for single datapair,\(\{K, S^t(K)\}_{t=1}^8\), we generate eight different eigenvector to better inform time dynamics of plume.

• Currently, the number of observation is 10. Increase to 100.

Evaluation: Posterior Estimate

\[ min_{K} \|S_{\theta}(K) - S(K*) \|^2_2 \]

where: - K0 = H(K) - S_{}: Neural Network model

Choosing the best parameters for MLE optimization

We conduct hyperparameter search for the \(\lambda\). The number of epochs chosen were based on the loss plot convergence. If it converged, we stopped training.

This is unconstrained. When lambda = 1., it is not accurate.

Unconstrained

	Epochs	\(\lambda\)	Loss (MSE)	SSIM
FNO-PBI	800	1.0	\(4.2823 \times 10^{-4}\)	\(63.8013\)
FNO-PBI	400	20.0	\(8.9021 \times 10^{-5}\)	\(55.5040\)
FNO-PBI	300	50.0	\(6.1867 \times 10^{-5}\)	\(55.5502\)
FNO-PBI	200	100.0	\(5.5757 \times 10^{-5}\)	\(57.0893\)
—————	———	———	————————	—————-
FNO-MSE	800	1.0	\(3.0692 \times 10^{-4}\)	\(57.3091\)
FNO-MSE	400	20.0	\(5.3901 \times 10^{-5}\)	\(52.3515\)
FNO-MSE	300	50.0	\(3.4602 \times 10^{-5}\)	\(54.2848\)
FNO-MSE	200	100.0	\(2.9429 \times 10^{-5}\)	\(55.6442\)

Updated Result

Originally, we wanted to use Normalizing Flow for our inference method. But because it takes quite a long time to train, for a quick evaluation, we first try least squares method. Out of all 200 test samples, I brought some interesting cases.

Test sample 1

Test sample 2

Test sample 3

Test sample 4

Test sample 5

Conclusion

• As of right now, we don’t see significant difference between MSE and PBI model in terms of posterior estimate.
  • It is likely undertrained.

Future Step

1. TODO: Debug NS eigenvector and vjp.
2. TODO: Want to generate the full dataset for Francis’ dataset (which might take 1 or 2 days).
3. TODO: Try it on Jason’s dataset (Now that we fixed the problem with FIM computation, we are optimistic about the experiment, so we want to try it again.)

Question

1. Do we want to train both models for a longer time?